Skip to main content

Full text of "ERIC ED216030: Reflections on the Movement to Promote Effective Educational Evaluations Through the Use of Professional Standards. Discussion Draft."

See other formats


DOCUMENT RESUME 



ED 216 030 

AUTHOR 
TITLE 



PUB DATE 
MOTE 



| TM 820 184 

Stuff lebeam, Daniel L. 

Reflections on the Movement to Promote Effective 
Educational Evaluations Through the Use of 
Professional Standards. Discussion Draft. 
Aug 81 

33p. ; Paper presented at the Annual Meeting of the 
American Psychological Association (Los Angeles, CA, 
August, 1981). 

EDRS PRICE MF01/PC02 Plus Postage. 

DESCRIPTORS Curriculum Evaluation; *Evaluation Methods; Program 

Evaluation; *Standards 
IDENTIFIERS *Standards for Evaluation Educ Prog Proj Materials 

ABSTRACT 

On behalf of the Joint Committee on Standards for 
Educational Evaluation, the author reviews the Committee's recent 
efforts and discusses the needs for further work related to the 
"Standards for Evaluations of Educational Programs , Projects, and 
Materials. " The Committee has developed a program to guide its work. 
The program addresses four areas of activity: provision of training 
and technical assistance; research, development, and training for 
interpretation and use of the Standards; dissemination of information 
related to the standards-; and revision of the Standards as needed. 
Critiques of the Standards appeared in the May 1981 issue of 
"Evaluation News." A content analysis of these critiques yielded both 
positive and negative comments in four areas: content, rationale, 
structure , and use. Further work on the content of the Standards 
should address the trade-offs among them, the use of needs assessment 
in evaluation, and audiovisual training materials. Dialogue among 
various standard-setting groups would further develop the Standards' 
rationale. The structure of the standard-setting process could be 
improved by expanding the Committee membership. The use of the 
Standards in field tests should be encouraged. (BW) 



********************************************* ************** ************ 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 
*********************************************************************** 



o 

K\ Discussion Draft 

O 

1—1 

CM 
Q 
UJ 



Reflections on the Movement 
to Promote Effective Educational Evaluations 
Through the Use of Professional Standards 



Presented by: 



Daniel L. Stufflebeam "permission to reproduce this 

Chairman of the Joint Committee on Standards material has been granted by 
for Educational Evaluation A^jALt L^ 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



X 

\ 

o 

N 

ft 



at 

the Annual Meeting of the American Psychological Association 



Los Angeles, California 



August, 1981 



OA. DCPAHTMCNT OF COUCAHOW 

NATIONAL INSTITUTE OF EOUCATION 

EDUCATIONAL RESOURCES INfORMATION 

CENTER <ERICI 
i/Th« document has been reproduced as 
recerved from the person or organization 
' ortgtnating it 

! Minor changes have been made to improve 
reproduction quality 

• Points of view or opinions stated m thrs.docu- 
ment do not necessanly represent otteial NtE 
position or policy 



ERIC 



1 

I am speaking to you today on behalf of the Joint Committee on Standards 
for Educational Evaluation., As you are aware, the Committee, last year, after 
nearly 5 years of work, Issued Standards for Evaluations of Educational Programs , 
Projects, and Materials - The Standards , as I will" refer to this publication 
hereafter in this report, is intended to provide evaluators and their clients 
with an authoritative guide for assessing and upgrading the quality of program 

o 

evaluations in education. 

At last year's APA meeting in Montreal, I reviewed the work involved in 
developing the Standards , and I described the Committee's plans for an ongoing 
program to promote the effective use of the Standards and periodically to 
accomplish needed revisions. This year I have been invited to bring you up-to- 
date on the Committee's recent efforts and specifically ;to discuss the needs 
for further work related to the Standards . I am pleased to respond to 
Dr. Diamond's invitation, because the Committee wants to continue tc be 
accountable to its Sponsoring Organizations, and because, as a volunteer group 
we need to involve as many interested persons as possible in our work. 

I have organized the body of my report into three parts. In the first 
part, I will review pertinent developments during this past year. Then I will 
identify and analyze issues concerning the Standards that I have drawn from a 
series of critiques of the Standar ds which appeared recently in Evaluation News , 
Finally, I will attempt to address the identified issues by proposing a partial 
agenda for conducting projects designed to improve both the contents and uses 
of the Standards . Without further introduction, I turn to my brief progress 
report. 



\ 

i 



1 . Progress This Past Year 

\ 

The Standards has been quite well received. At last report about 4,000 
copies had been sold and McGraw-Hill was in the process of ordering a second 
printing. 

The Committee and its Sponsoring Organizations have taken several steps 
, to meet the needs for an ongoing standard-setting effort. The Conmittee 
developed a plan for a standing committee, developed a set of Principles and 
By-laws to guide and govern the work of the new Committee, and became a 
legally constituted body. 

Eleven organizations decided to sponsor the continuing effort; and each 
one appointed a representative (See Exhibit 1) to serve on the Committee and 
agreed to cover the member's expenses associated with the Committee's annual 
meeting. The new representative from APA is Dr. Carol Kehr Tittle. The 
Committee arranged to use the royalties from the sales of its publications to 
help support its work; while minimal, these funds provide a measure of ongoing 
support. The Committee also obtained grants totalling about $6,000 from the 
EXXON Foundation and the International Paper Confpany Foundation to support 
basic planning and organizing activities. Currently, the Committee is develop- 
ing funding requests for a number of specific projects. 

The Conmittee has also begun to respond to a number of inquiries and 
requests for assistance. For example, individual members have reacted to the 
efforts of personnel of the Louisiana Department of Education to apply the 
Standards on a statewide basis; members have conducted workshops on the Standards 
in Malaysia, Australia, and New Zealand, as well as the U.S., and personnel of 
the Evaluation Center have responded to numerous requests for information. Also 
The Committee has collaborated with the ERIC Center at ETS toward the development 
and publication of an annotated bibliography to accompany the Standard s. 



In addition to the preceding specific steps, the Committee has defined a 
general program to guide its work. This program is divided into the following 
four areas: 

1. Provision of training and technical assistance to evaluators 
and users of evaluations. 

2. Research, development, and training associated with interpreting 
and using the Standards . 

3. Dissemination of and clearinghouse for information related to 
the Standards and their use; and 

4. Revision of the Standards as needed, and expansion of their use 

in new directions (e.g., evaluations of personnel and facilities). 

The preceding report of activity demonstrates that the Committee has laid 
a substantial foundation for an ongoing effort to develop and promote sound 
use or evaluation standards. Next, the Committee must begin implementing their 
program by choosing arxTcarrying out specific projects that respond to high 
priority needs. 



. -4- 

2. Analysis of Issues From the Literature 

One basis for identifying these needs is to be found in the May 1981 
issue of Evaluation News , That issue contains' a criticajl appraisal f of the 
Standards > The 10 articles that convey this appraisal include the following: 
general reviews by Sechrest and Stake; implications of the Standards for 
licensure and accreditation by Becker and Kirkhart; Federal and State perspec- 
tives, respectively, by Wargoand Baron; an urban school district perspective 
by Osterlind; a rural school perspective by Hecht; an out-of-school learning 
perspective by Marcia Linn; an international perspective by Searle; and a 
mental health programs perspective by Lyons and Rubin- Collectively, these 
articles provide a rich and varied view of the Standards , 

The Joint Committee has always believed that one of its main responsi- 
bilities is to seek out, study, and react to critical appraisals and field 
tests of its products. Such evidence provides a vital basis for ensuring 
that the Committee's work is sensitive and responsive to what persons in the 
field see as real and important evaluation problems. It is fitting, then, 
that the Joint Committee seriously study the reactions to their work that 
appeared in the May, 1981 issue of Evaluation News . 

In order to nine the articles for potentially useful leads, I performed* 
a content analysis across all eleven of them. First, I made a list of allegations 
about the Standards from each article (See Exhibit 2). In accordance with the 
views presented by the authors, I divided the allegations into lists of strengths^, 
weaknesses, and suggestions. Then I sorted the items in each list into four 
categories that were previously found to be useful for organizing research and 
development activities in relation to the standards for accounting and auditing. 



ERIC 6 



As reported by Ridings , these categories are: 1) the rationale for stating 
and using standards in a field; 2) the content of the standards; 3) the structure 
by whid/the standards are set, reviewed, and modified; and 4) uses of the 
standa/ds . The result of these sorts was a matrix of strengths, weaknesses, 
and ^commendations in each of the four categories (See Exhibit 3). 

This matrix revealed a number of general characteristics of the eleven re- 
views. On the whole, the critics found the Standards to be responsive to a 
real need and of high quality. In a number of respects, however, they pointed 
to needs for further testing and revision. Occasionally, what one critic saw 
as a strength was viewed by another as a weakness; for example, Wargo applauded 
the specific advice in Standard Al regarding rank ordering of audiences, while 
Stake cited this an an example of overspecif? cation. Generally, it seemed 
clear that all of the critiques were armchair evaluations, since none of them 
identified any strengths and weaknesses as regards actual uses of the Standards . 
But a number of the authors did recommend that the Standards be field tested 
in a wide range of settings. Clearly, the observations made by these authors 
should be viewed as tentative and subject to verification, and not a sufficient 
guide for revising the Standards and the standard setting process. 

Nevertheless, the matrix of information does point up a number of problems 
that might be addressed by the Committee or other groups. I shall mention a 
few of these to give you a general idea of the needs identified by the critics. 
I'll discuss each of the four categories in turn. 

The most comments were contained in' the category which concerns the 
content of the Standards. These comments reflected extensive praise of the 



1 J 

Ridings, Jeri M. Standard Setting in Accounting and Auditing : 
Considerations for Educational Evaluation- Kalamazoo, Michigan: Western 
Michigan University, 1980 (unpublished dissertation). ; 



/ 

/ 

document. Among the positive statements were the following: <" 

--the advice in the Standards is realistic, practical, usually 
of admirable temperament, and commending of adherence, 

—the Standards rs aimed at a varied audience and provides a 

comprehensive, .useable, and "concise" statement of good 
practice. 

—the scope of the Standards is thorough, and the document 

- raises a large number of key issues, 

- offers many good suggestions.* 

- endorces methodological variety* 

- focuses on a variety of evaluation situations, 

- provides a range of interesting and realistic , 

illustrations; and 

- confronts a range of political realities. 

Despite the overall positive assessment of the content of the Standards , 
the authors also offered a number of criticisms. Several ofjthe writers 
pointed to the problem of conflicts between different standards and observed 
that the Committee hadn't offered enough useful advice about how to identify 
and address- problems involving trade-offs among the standards. Whereas there 
were many compliments about the scope of the Standards , the document was also 
criticized for being less than comprehensive. Stake, for example, noted that 
the Standards is incomplete in its identification of instances of malpractice ' 
that are common in the educational evaluation establishment, such as promising 
what can't be done. Several writers also observed or implied that the 
Standards should explicitly require that needs assessment be a part of every 
evaluation. Sechrist criticized the Committee for not including a standard 
requiring the best applicable research design. The essence of these 
criticisms is that the scope of topics treated and practices required should 
be expanded. 

Other criticisms concerned the useability of the Standards , They were 
seen not to reflect considerations found in rural settings, Federal evaluations, 

and evaluations in foreign countries. 

/ 



-7- 

In general, the criticisms of the contents of the Standards was construe- 

j tiye and pointed to possibilities for improvements. This is borne out by the 

" many recommendations that the reviewers offered. They called for companion 

volumes to help various groups use the Standards . They suggested adding 

new standards and other illuminating material in the next edition of the 

Standards . And' they suggested how several of the present 30 standards might 

bemodified. I have made an inventory, of these suggestions and will present 

them for consideration and further study at the next meeting of the Joint 

Committee in October. 

Turning frotii the content of the Standards , the eleven articles I 

examined also provided insights concerning the rationale for an effort to 

set and use standards in educational evaluation. As Ridings pointed out, 

it is fundamentally important that standard setters periodically review the 

role of standards in their profession in order to ensure that positive ends 

are being served and that negative side effects are being minimized. Stake , 

granted that the rationale for standards has not been put better than by the 

Joint Committee/ Other writers observed that the Standards likely would have 

many positive benefits. :> 

These include providing * 

--benchmarks for judging evaluations 

--content for inservice and preservice training 

—a guide for developing and licensing training programs 

--a guide for developing standardized tests on evaluation 

--a framework for a competency-based approach to accrediting 
and licensing e valuators 

--a framework for developing evaluation' contracts 

—a stimulous for better conduct and use of evaluation; and 

--encouragement for improving the integrity of evaluators 

ERIC , 9 



While some of these purposes might be controversial, on the whole they present 
• * 

a positive view of the potential effects of standards for evaluation. 

On the negative side of the ledger, Stake saw some problems in how the 
Committee had addressed the issue of a rationale for standards in evaluation. 
He said that whereas the Committee had made a strong case for standards, 
they hadn't been sufficiently vigorous in examining the case against standards . 
He said that standards must certainly constrain Nativity, and he speculated 
that they might unjustly deny to individuals the privilege to practice 
evaluation. 

If I may be permitted to editorialize on this point, I think the Joint 
Committee would have me emphasize that they see 'the Standards serving more 
to promote and aid quality evaluation than to constrain or punish evaluators. 
Nevertheless, I must acknowledge that at least one department of education 
is attempting to use training in the Standards as one basis for granting 
the right to practice educational evaluation. This comes close to illustrating 
what Stake sees as a potential debilitating effect of the Standards . Since 
otherwise highly qualified evaluators might be prevented from practicing 
evaluation in the state simply because they hadn't attended the designated 
training session. Clearly the Joint Committee must be vigilant and proactive 
in this leadership towards helping the field to make sound use of the 
Standards and avoid some of the negative possibilities. 

Another issue regarding the rationale for standards was pointed up in 
the article by Lyons and Rubin. They referenced about half a dozen standard 
setting efforts that have relevance to educational evaluation. "Ridings and 
other wirters have pointed* out that multiple stagdard setting pr<v- ams in a 
given field create competing .efforts that have the potential to confuse clients, 
waste resources, and fragment professional development. However, Lyons and 

10 



Rubin argued that evaluators' needs in different contexts are quite different, 
and they saw the creation of multiple sets of standards as potentially more . 
responsive to idiosyncratic needs than would be the case with a single set 
of general standards. Stake agreed with this position when he called for 
different sets of standards that reflect local norms and conditions^ 

On this point, it seems clear that the Joirtt Committee end other standards 

# 

setting groups must carefully consider how they can collectively and individually 
best serve the needs of education. At the very least, the Joint Committee and 
the body responsible for the ERS standards should increase their dialogue 
about possibilities for collaboration. 

This point is a good one for a transition to the matter of the adequacy 
of the Joint Committee's structure for setting standards. Given that 
standards for evaluations are needed, and given that their acceptability to 
the field depends on the credibility of the process by which they were 
developed, then it is important that the Joint Committee periodically review, 
-and, where necessary, revise the structure of their standard setting process. 

In general, the authors of the articles on the Standards which appeared 
in Evaluation News observed that the developers of the Standards possessed 
high credibility. They also acknowledged that the Joint Committee made good 
use of a large and diverse number of advisors. - % 

However, several of the writers pointed to deficiencies in this area. 
Especially, they said that the developers and support groups underrepresented 
certain groups. These included federal agency personnel, contract research 
organizations, rural school personnel, and foreign groups. The general point 
is well taken. I believe the Joint Committee continuously must attempt to 
ensure that all stakeholder groups are adequately represented in the standard 
setting effort. 

t 

li . o 



How to do this leads us to our final area of concern regarding the/ 
Standards ; this is the matter of Uses, and I ought to add, Misuses of ttw 
Standards . I've already noted that none of the eleven articles being 
discussed made any reference' to actual uses of the Standards . Howler, 
they did offer recommendations in this realm. Wargo emphasized, 'for example, 
that the standards* should be field tested at the Federal level, as a basis 
for revising them, so that they would be more -useful to federal level . 
evaluators—Marcia Linn suggested- that the standards be field tested in out- 
of -school settings and in a. variety of other "very different" settings such 
as other countries; she thought sudh tests would be very instructive as 
regards limitations and needs for revisions in, the present Standards . Hech- 
added that the Committee should respond to the needs of rural school 
personnel by disseminating the Standards through non-print media. Finally, 
Stake implied that the Committee has a responsibility to help the field 
avoid uses of the Standards that constrain individual practice. 

The preceding comments about all four aspects of the Standards being 
considered—i.e. , fcbnteot, rationale, structure, and use--have direct 
relevance to the future work of the Joiht Committee. Overall, these comments 
indicate that the work of the Committee, to this point, is appreciated, 
respected, but not yet -complete. These are recommendations for improvements 
and extensions, and they underscore the importance of the Committee's future 
role as regards improvement of the contents and uses of the Standards . Also 
there is clearly a need to provide strong leadership to ensure that the standard 
setting efffort will continue to serve worthy purposes. There is a need- to 
increase/the effort to ensure that all stakeholder groups are properly involved 
and served. And there is definitely a need to obtain empirical evidence about 
uses of the standards. 



. 12 



-11- 

3, A Proposed Agenda of Needed Work 

How to address these four areas of need is the topic with which I will 
close this paper. Specifically, I will propose an agenda of activities. This 

agenda is intended partially for the consideratioaof— the_Joint Commi±tee__as 

they plan and seek fiinds to support their work. But it is also intended for 
researchers and developers who are interested^ the area of evaluation. 

The agenda that I would have us consider Erectly reflects the 
preceding Analysis. I have identified what I consider to be one or more 
high priority projects in each of the four areas- -content, rationale, 
structure, and use. In selecting thSse projects I have attempted to respond 
to the information contained in the eleven Evaluation News articles. The 
proposed projects mainly reflect an action research agenda. 

In the area of content, I have three projects in mind. These are: 
1) identifying and dealing with trade-offs among the standards; 2) developing 
and testing a standard that requires the use of needs assessment in evaluation 
"an<k3) developing audio visual materials to assist trainers in training non- 
technical ly oriented audiences to understand and use the Standards . 

Regarding the rationale for Evaluation Standards, I believe the time has 
come for dialogue among the various groups which are involved in setting 
standards for evaluation. A well planned conference involving the representa- 
tives of these groups would clearly be in order. The conference could address 
questions concerning the relative merit of unified versus diversified standard 
setting efforts. It could also search out and promote areas for future 
collaboration. I think the participants in such a conference should represent 
the ERS and Joint Committee standard setting groups, as well as other relevant 
groups, such as the Committee on Standards for Educational and Psychological 
tests. Moreover, the conference group definitely should cut across national 
boundaries. 

ERIC 13 



-12- 

In the third work area— that is concerned with the structure of the 
process— I believe the Joint Committee must take concrete steps to respond 
to the criticisms th<at were offered by Wargo, Searle, and Hecht. Considera- 
tion should be given to expanding the membership of the Joint Committee, so 
asn&etteFTo ref lect~pefspei:tlves~that are not represented now. Also the 
Committee should consider other means of increasing communication such as 
review panels, and a newsletter. 

Finally, as regards use of the Standards I believe the Committee should 
encourage field tests of the standards in a variety of settings. Especially 
they should promote the conduct of case studies in other countries, at the 
Federal level in our country, in contract research corporations, in rural 
schools, and in out-of-school settings. The Committee should arrange to use 
the case reports in revising the Standards and should publish them in a form 
designed to help users of the Standards to see them in contexts that have 
particular meaning to them. 

The projects I have just referenced, of course, are incompletely 
described and must be deliberated by the Joint Committee and its constituents. 

In presenting them I have tried to show the kinds of work that might be done 

i 

to improve the content and use of the standards for evaluations. I have also 
tried to illustrate that feedback from the profession is a vital means of 
determining what needs to be done. 



ERLC 



14 



-13- 



Exhibit 1 

THE JOINT COMMITTEE 
(As of July 1, 1981) 



Esther Diamond (Science Research Associates), representing the Association 
for Measurement and Evaluation in Guidance 

Roy Forbes (Education Commission of the States), representing the Education 
Commission of the States 

Freda Holley (Austin, Texas Independent School District), representing the 
American Educational Research Association 

Philip Hosford (New Mexico State University), representing the Association 
tor Supervision and Curriculum Development 

William Mays, Jr . (Michigan Elementary and Middle Schools Principals Association), 
representing the National Association for Elementary School 
Principals 

Bernard McKenna (National Education Association), representing the National 
: Education Association 

Lloyd Nielsen (Roseville, Minnesota School District), representing the 
American Association of School Administrators 

James Oglesby (Columbia, Missouri Public Schools), representing the National 
School Boards Association 

James Sanders (Western Michigan University), representing the Evaluation 
Network- 

Daniel Stuff! ebeam (Western Michigan University), representing the National 
Council on Measurement in Education 

Carol Kehr Tittle (University of North Carolina at Greensboro), representing 
tne American Psyc jlogical Association 



9 

ERLC 



15 



Exhibit 2 
Critical Reactions to the Standards 

•s 



) 


' Pros 


Cons 


Suggestions 


Lee Secbrest 


—Benchmark for judging 

evaluations 
—Content for training 
—Useful Inventory of 

Issues 
—Credible developers 
—Consistent format 

amendment 

—Realistic and comman- 
ding of adherence 

—Interesting and real- 
istic illustrations 

—Potentially useful 
for contracting. 


—Number (too many to 
live up to) 

—Incompatibilities 
(trade-offs) 

—Despite advice in 
Audience Identifi- 
cation Sometimes 
it is legitimate to 
ignore an audience 

—Lack of specific 
reference to appro- 
priate experimental 
designs. 


—Add a standard calling for 
only valid and dependable 
designs, (best possible 
research design) 


Robert Stake 


—Case for standards 
has best been stated 
by the Committee 
(p. 149) 

—Standards that can't 
be measured are okay, 
because they arouse 
people to care about 
what someone acroi res 

—Standards raise a 
large number of 
important issues 

—Most of the statements 
are of admirable 
temperament. 


—Committee didn't 
vigorously examine 
the case against 
the standards. 

--Standards bring 
injustice and con- 
straints on 
creativity 

— — OLCuiuaiUo will a ! u 

unjust denials of 
the privilege to 
practice evalua- 
tion. [We already 
see this in 
Lousisana], 


--Avoid constraining individual 
practice if done only to pro- 
tect our professional way of 
life. 



V 



, 17 

erJc 16 



Exhibit 2 (continued) 



9 

ERIC 





r i ub 


\ f nnc 


ouyycb 1 1 uiib 






1 

ooiiic ui tnc buanuarub 






»» 


IP n aii/Hpnr*P "iHpn — 
« \ c • y • , auuiciiuc i ucii 






■ 

1 


tification are over- 








specified) 


t 






—Standards haven 1 1 


--Implies a need for standards 






luciicii icu iiiaiijr uau 


fhaf nrnhiM^ unrpal i^'Hr 






nrarHrPQ f hv "fillANGfi 1 ' ) 


nrowicp *hpinn indi'ff prpirf ^o 
ui mu i oc , uc i iiy i nu i i i ci en 






so tacitly legitimate 


concerns of evaluees, and pro- 






fhom /n "I \ 


mUCc cTTUI tb tu uriUcibUaiiU 






education. 








-_D**nv/"i rlo cfanHa **Hc h\/ pyamnlpc 








in ntir mpta-Pvaluat"ion^ a<^ an 

ill uui iuc v*o cvu luav* i uiij u o on 








alternative to formal standards 

U 1 vvl MUv 1 IV W \J 1 VI IIIU 1 J WUIIUMI U«l 


Hpafhpr Rprkpr 


Thp ^frandard^ nrovidp 


\ 




anu 






V 


Karen Kl rKRaru 


Diny wjUcbtlUMb per" 








^a4n{ nn ^n Houolnninn 
Laiiiifiy lu ucvciupiny 








3 rnmnp^pnrv-ha^pd 








annpnarh tc\ nrrrfrd'it - 








4 ml anH 1 i rone i nn 

iny ana i icciibiiiy 








cVa I UalUi b • 








- Prnvf Hp a no^i hi p 








ntHrlp "fnv* dpvplnn— 








iny anu clcli eui tiny 








train iny pray rains 








• Prnvidp a nn^<H hip 








ha c^c *f n** Hpv/p Inn — 
Dab lb TUT UcVclUU— 








iny a SianuarulZcu 








Lcbt TOY abbebb lily 








onp's knowlpdap of 

\J \ 1 w w l\Mvn 1 CU^C \J \ 








evaluation models, 








techniques, and 








principles 








- Guidelines could be 








used to develop be- 








havioral criteria for 






18 


judging an evaluator's 




19 


performance. 





Exhibit 2 (continued ) 





Pros 


Cons 


Suggestions 


Heather Becker 
and 

Karen Klrkhart 


—Scope of the Standards 
is most thorough (and 
are sufficient in 
breadth to serve as a 
basis for licence 
(accreditation). 


--Exclusion of needs 
assessment 

—But they don't help 
with the choice of 
those that are most 
relevant for licence 
and accreditation 


--Propriety and accuracy are 
viewed as most important in 
accreditation an<* licensing 

—Suggests possibility of a 
Consumer's Guide to Program 
Evaluation as a companion to 
the Standards, 

--The Standards require fine 
tuning, discussion, and 
negotiation before the Standards 
could be used for licensing based 
cn criteria measures • (many 
issues must be resolved) 


Michael J, Wargo 


—Provide a comprehensive 
yet consise statement 
of "good practices" 

—Good organization and 
format 

—Functional reorganiza- 
tion useful 

— Endorcement of methodo- 
logical variety 

—Emphasis on evaluation 
quality 

—Rationale for'standards 

—Focus (small-large, etc,) 

—Confronts political 
realities 

—Targeting at varied 
audience 

—Agrees with need to 
rank order audiences 


— Underrepresents 
Federal level groups 
on Jt. Comm. and 
support groups 

--Underrepresents 
contract research 
organizations 

—Unrealistic to assume 
evaluators are suffi- 
ciently independent 
to be able to quit 

—Standards fail to 
recognize and deal 
with the complexities 
of team work involved 
in federal evaluations 

—Most case studies are 
at state and local 
level 

--But says information 
needs within audiences 
must also be ranked 


—Recognize significant differences 
between federal and nonfederal 
evaluations, re: i, 

- audiences/stakeholders ? 

- contextual influences 

- number of data sources 

- distances 

- costs 

- constraints 

--Standards must be field tested and 
reviewed at federal level and re- 
vised accordingly (need case his- 
tories of successes and failures) 



ERIC 



Exhibit 2 (continue d) 





Pros 


Cons 


Suggestions « 


Michael J. Wargo 
(continued) . 




--But says informations 
needs within audiences 
must also be ranked 

--Says cost/effectiveness 
standard overlooks dis- 
tribution of costs and 
benefits across differ- 
ent levels of program 
operation and audiences 


—Suggests that Al give more 
specific advice about rank- 
ing of audiences 

—Suggests the A4 be reworded 
and given emphasis on compara- 
tive cr discrepant- nature of 
most evaluations ■ x - 

—Says A8 needs to go further, - 
re: ensuring impact 

—CI should recognize multiple 
clients and calls for a new 
guideline on this 


Joan Boyhoff Baron 


—Practical orientation 

—Potential to improve 
quality and use 

—Requirements of care- 
ful descriptions good 

--Cases provide good 
examples of *hy program 
description is impor- 
tant 

—Good advice, re: search- 
ing through existing 
information 

—Greatest strength is in 
sensitizing people to 
consider all key 
audiences 

—Use of a large number 
of advisors in develop- 
ing the standards 

—Open methodological 
base 

--Short, readable 


--They lack the context 

needed by the unitiated 
--Standards mean "how 

much is enough". 

These standards aren't 

standards. 


--State officials should haye 
several copies for use and 
circulation 

—But should be extended to * 
cover proposals for state fund- 
ing (object description is also 
important there). (Describe 
from perspective of the client) 

—Give more visibility to the 
"forming of advisory groups." 

--Give attention to issues in 
choosing an internal or 
external evaluator 

—Emphasize the importance of 
determining clients 1 needs 

—Add concern for stating the A 
limitations of an evaluation 
in the final section of every 
evaluation report 

— Relable the Standards as 
principles, practices, or 
methodological guidelines. 



22 



23 



Exhibit 2 (continued ) 





Pros 


< 

Cons 


Suggestions 


Steve J. Osterlind 


— Dellnatlhg criteria 
encourages Integrity 
of program eva.luators 
(the other side of 
State's notion that 
omlting bad practices 
supports disservice) 

—Provides direction for 
Inservice programs 

—Provides key check- 
points for determining 
competency of prospec- 
tive employees 

—Could foster a commun- 
ity spirit and bench- 
marks for judging 
quality evaluations 

—Responsive to field's 
need for structure 
and guidelines as 
ODDOsed to orioinal 
methodologies 

—Functional table is 
highly useful. 


* 

< 




Kathryn A- Hecht 


— Standards Intended 
for a wide audience. 


--Style and volume 
militate against use 
by lay audiences 

--But they're too con- 
denced and distil Ted 
to be o f use by the 
uninitiated. 


—Develop and diversify for use 

with multiple audiences 
—Adapt to rural needs and 

audiences 
—Develop guidelines on how to 

integrate Into evaluation 

process 

—Disseminate through non-print 
and print media. 



25 



Exhibit 2 (continued ) 





Pros 


Cons 


Suggestions 


Marcia Linn 


—Plethora of sound 
suggestions* 


--Identify important 
elements more effec- 
tively then they 
characterize* 
trade-offs 
Tell more about what 


—Augment standards with specific 
knowledge of the area being 
investigated and with other con- 
ceptual frameworks. 






not to- do than clever 
tfiTngs that, could be 
done (e.g. , resist 
pressures and don't 

May thus inhibit 
evaluator and reduce 
his power. 




Barbara Searle c 


• 


--American setting is 
less centralized, more 
affluent and utility 
and propriety standards 
won't cpply as well 

--No guidelines for rank 
ordering the Standards 

--Provinci al . 

1 1 J v 1 IIV 1 U 1 . 


t 

--Interpret them in educational 
settings, very different from 
those for which they were 
developed in order to increase 
understanding of them and- 
improve usefulness. 


Ray Lyons and 
Bill Rubin 


—Cases are extremely 
informative. 




—Development of supplementary 
book on implementation ideas 
is a good idea 

--Sees variety of models and 
standards to be desireable. 



27 



Exhibit 3 



Content, of the 
Standards 



9 

.ERIC 



28 



An Interpretive Analysis of the Reactions to the Standards 



Pros 



-Advice is realistic, 
practical, usually of 
admirable temperament, 



and commanding of 
adhei .nee 

-Scope is most thorough 
and the Standards : 

- raise a large num- 
ber of key issues 

- offer many good sug- 
gestions 

- endorce methodolo- 
gical variety 

- focus on a variety 
of evaluation 
situations* 

- provide a range of 
interesting and 
realistic illus- 
trations 

- confront a range of 
political realities 

■ Standards is aimed at 
a varied audience and 
they: 

- provide a comprehen- 
sive yet concise 
statement of "good 
practice" 

- are short and 
readable 

- provide an organiza- 
tion and format which 
is easy to use 

- offer a useful func- 
tional table of 
contents 



Cons 



-Some standards are 
incompatible, but: 
- trade off problems 



are not we W d escribed 

- guidance for rank 
ordering standards is 
too general 

-Scope is too limited in: 

- identification of bad 
practices in the eval- 
uation establishment 

- treating needs 
assessment 

- suggesting clever 
tactics 

- not requiring experi- 
mental designs 

- Standards are difficult 
to use because they: 

- are numerous 

- lack enough context to 
help the uninitiated 

- are not ranked for 
their relevance to 
licensing and 
accrediting 

- are provincial 
■ Standards have limited 
use at Federal level 
because they? 

- assume, unreal istical - 
ly, that evaluators 
can quit 

- fail to recognize and 
deal with the compl ex- 
it! es of team work 
involved in Federal 
evaluations 

- include too few case 
studies at the 
Federal level. ■ 



Recommendations 



-Develop companion .volumes to in- 
clude: ; 

- a sunmplemental book on imple- 



menting the Standards 

- a consumer's guide to program 
evaluation 

-Consider adding materials that: 

- prohibit unrealistic promises 

- prohibit being indifferent to 
the concerns of evaluees 

- promote efforts to understand 
education 0 

- require specific knowledge of 
area being investigated 

- encourage use of a variety of 
conceptual frameworks 

- call for only valid and defen- 
sible designs 

- project propriety and accuracy 
standards as the most important 
in accreditation and licensing 

- give more visibility to form- 
ing of advisory groups 

- provide guidance for choosing 
internal and external evalua- 
tors 

- emphasize the importance of 
determining clients 1 needs 

- suggest stating limitations 
of the evaluation in the final 
section of the evaluation report 

- show how to integrate the 
standards in the evaluation 
process 

- show how to employ the 
standards in licensing 



29 



i 

o 
i 



Content of the 
Standards 
(continued) 



Rationale for 
Developing and 
Using Standards 



30 



Pros 



Exhibit 3 (continued ) 
Cons 



-Particularly helpful 
suggestions Include: 
- advice on rank order- 
ing audiences 
requirements for car e- 



ful descriptions 
- advlfce on searching 
through existing 
Information 
-Standard s both emphasize 
quality evaluation and 
respond to the fields 
need for structured and 
concrete suggestions as 
opposed to offering 
vague suggestions for 
using "original' 1 
methods. 



-Case for standards has 
best been stated by the 
Committee 

-The Standards have many* 
potential benefi ts : 

- benchmarks for judging 
evaluations 

- content for inservice 
and preservice train- 
ing 

- guide for developing 
standardized tests on 
evaluation 

- framework for a* com- 
petency based approach 
to accrediting and 
licensing 

- framework for develop- 
ing evaluation con- 
tracts 



—The Audience Identifica- 
tion standard was 
claimed to be both: 
- overspedfied 

— - unde r specif i 

—The- Cost Effectiveness 
standard overlooks the 
distribution of costs 
and benefits across 
different levels of 
program evaluation. 



-Committee wasn't suffici- 
ently vigorous in exam- 
ining the case against 
the standards 

-Standards may aid unjust 
denials of the privilege 
to practice evaluation 

-Standards constrain 
creativity (Stake) 

-Committee's definition 
of standards may be 
to general 



Recommendations 



-Diversify the Standards for use 
with: v> 

- federal audiences 9 

- rural audiences 



-=rAdd specificity in Al reranking 

of audiences 
—Emphasize in A4 that most 

standards are comparative or 

discrepant 
—Expand A8 to promote more Impact 
—CI should recognize multiple 

audiences 
—Consider relabeling the Standards 

as Principles, Practices, or 

methodological guidelines 



--Provide standards by example in 
meta-evaluatlons as an alterna- 
tive to formal standards 

—Leave room for a variety of 
models and standards in order 
to accommodate different 
settings. 



31 





Exhibit 3 (continued) 


% 




Pros 


Cons 


Recommendations 


Rationale for 
1 Developing and 
. Using Standards 
(continued) 


- promotion ot oetter 
conduct and use of 
evaluation 

- encouragement for 
improving the integ- 






- 


rety of evaluators ■ 
—Standards that can't be 

mpAQiirPrl arp okav 
iiicodui cu ai c vi^ajf 

because they arouse 
people to care about 
what others admire. 






Structure of the 
Standard Setting 
Effort 


--The developers of the 
Standards possessed 
high credibility 

—The Committee made^ 
good use of a large 
number of advisors. 


—The developers and 
stpport groups under- 
represented certain 
groups including: 

- federal agency 
personnel 

- contract research 

- foreign groups 

- rural school 
personnel 


--Future developmental efforts 
should increase the involve- 
ment of the previously under- 
represented groups. 


Uses of the 
Standards 

e 






—State, agency personnel should 
maintain a supply of the 
Standards for distribution to 
tlie groups they serve, 

—Resist use^of the Standards 
that constrain individual 
practice if done only to pro- 
tect the evaluation profession 

--The Standards should be field 
tested at the Feaeral level 
and *7) a range of very differ- 
ent settings 

—the Standards should be dis- 
seminated through the use of 
non-print media* 


ERJC 32 






33